Feat: to_json Infinity/-Infinity Nan values support#3875
Feat: to_json Infinity/-Infinity Nan values support#3875kazantsev-maksim wants to merge 61 commits into
Conversation
This reverts commit 768b3e9.
|
@kazuyukitanimura @andygrove Could you take another see when you have time. |
| -- query expect_fallback(Struct type: StructType(StructField(a,ArrayType(IntegerType,false),false)) contains unsupported types) | ||
| query ignore("Need support Spark 4.0.0") | ||
| SELECT to_json(named_struct(a, array(b))) |
There was a problem hiding this comment.
could you create a separate structs_to_json_spark4.sql for these tests and set min spark version 4 for that file, and change these to expect_fallback
There was a problem hiding this comment.
I updated the branch from master and updated the tests.
…support # Conflicts: # spark/src/main/scala/org/apache/comet/serde/structs.scala
|
I started CI. I will review once CI has run. Thanks @kazantsev-maksim |
|
Tests using expect_fallback work incorrectly on Spark versions >= 4.0 |
|
|
||
| statement | ||
| INSERT INTO test_to_json VALUES (1, 'hello'), (NULL, NULL), (0, '') | ||
| INSERT INTO test_to_json VALUES (1, 'hello', cast('NaN' as float), cast('Infinity' as double)), (NULL, NULL, NULL, NULL), (0, '', 0.0, 0.0) |
There was a problem hiding this comment.
Could you add -Infinity to this test
| .newBuilder() | ||
| .setChild(p) | ||
| .setTimezone(expr.timeZoneId.getOrElse("UTC")) | ||
| .setIgnoreNullFields(true) |
There was a problem hiding this comment.
Should setIgnoreNullFields be hard-coded to true?
There was a problem hiding this comment.
Also, the native code seems to ignore this field?
There was a problem hiding this comment.
I think we need to respect SQLConf.get.jsonGeneratorIgnoreNullFields
3 participants
Which issue does this PR close?
Closes #3016
Rationale for this change
What changes are included in this PR?
How are these changes tested?
Tested with existing unit tests